Many articles, of which documents are one type, are created based on information contained in a database. Such articles have data that remains constant and other variable data obtained from the database. For example, there may be a set of documents each containing a list of items offered for sale. All of these documents contain header data and general sales terms, for example, which would remain constant. The data for each individual item, such as product category, product description, product image, price, and SKU number, is variable and would differ for each individual item. The Internet, for example, contains many web pages similarly constructed—where several web pages have similar or identical constant data and variable data associated with a variety of information fields.
It is desirable to extract this unstructured variable data contained in the documents so that the data can be structured. Providing structure to this variable data allows the data to be more easily searched, presented and processed more efficiently than could the original documents. For example, the Internet contains a great number of web sites offering items for sale. However, a user who wishes to purchase a particular item from the Internet conventionally visits the web sites offering the item for sale to compare the sales terms of the item, such as price. Such an approach can be very time consuming. It is, thus, desirable to extract the relevant data about the item from all or several of the websites offering the item for sale and present it to the user in a single web page or a series of web pages.
In situations such as the Internet, web pages are formatted for viewing by people. Due to the unstructured nature of web pages, the variable data is often embedded with other format and constant data making the identification of variable data difficult. The owner or author of the web pages may not be the entity that desires to extract the data. This results in problems when trying to extract the variable data.
Some methods and systems exist that attempt to identify, extract and manipulate this variable data. Most of these methods and systems utilize a wrapper procedure. A wrapper uses the formatting conventions of a document to identify the variable data. For example, if one wants to extract price information about items for sale from a document and the item names always appear in bold and the prices always appear in italics in the document, a wrapper can be created to recognize this format and extract the data identified as bold as the item names and extract the data identified as italics as the prices.
Some systems and methods use a customized wrapper procedure. Because few document and web page authors publish their formatting conventions, a designer of a customized wrapper must manually construct a wrapper for each type of document. Further, the format of a document may periodically change. As a result, customized wrappers are tedious and error prone.
Other systems and methods attempt to learn wrappers from labeled examples of the data required to be extracted from documents. One such example is described in Kushmerick, N., Wrapper Induction: Efficiency and Expressiveness, Artificial Intelligence J. 118(1–2):15–68 (2000) (special issue on Intelligent Internet Systems), and http://www.cs.ucd.ie/staff/nick/home/research/download/kushmerick-aij2000.pdf. The wrapper induction technique described by Kushmerick involves learning the formatting conventions of a web page from a set of examples of a resource's web pages, each annotated with text fragments to be extracted. The Kushmerick technique typically requires a user to point out examples of the types of fields to extract data from before the wrapper induction is run on a page and the technique has problems properly identifying variable data if the format changes slightly from web page to web page.